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Overview  of  project:  The  original  goal  of  this  project  was  to  develop  GPU-based 
approaches  to  electronic  structure  and  ab  initio  molecular  dynamics  and  to  apply  these  to 
problems  in  mechanochemistry  and  photochemistry.  The  promise  of  the  graphics 
processing  unit  (GPU)  in  quantum  chemistry  and  ab  initio  molecular  dynamics  has  been 
largely  realized  and  as  a  consequence  of  this  NSSEFF-funded  research,  it  has  now  become 
routine  to  simulate  molecules  with  hundreds  of  atoms  for  time  scales  of  over  100 
picoseconds  from  first  principles  (using  only  a  single  desktop  machine  in  days).  These 
simulations  include  all  electrons  treated  quantum  mechanically,  and  therefore  can  treat 
bond  rearrangement,  electron/proton  transfer,  and  electronic  excitation  without 
constraints  or  foreknowledge.  This  new  capability  is  revolutionizing  molecular  simulation 
and  we  have  used  it  to  study  molecular  behavior  after  photon  excitation  and/or  under 
mechanical  force.  Below,  we  detail  some  of  the  advances  in  GPU-based  first-principles 
molecular  simulation  achieved  in  this  project.  However,  we  also  stress  that  there  were 
three  new  areas  which  were  incubated  by  this  grant  -  advances  which  could  not  be 
anticipated  when  the  grant  was  awarded,  but  which  are  themselves  revolutionary.  These 
are  1)  the  development  of  tensor  hypercontraction  which  exploits  ideas  from 
recommendation  systems  (as  used  by  Netflix  and  Amazon)  to  decrease  the  scaling  of 
wavefunction  methods,  2)  the  introduction  of  interactive  ab  initio  molecular  dynamics, 
where  virtual  reality  interfaces  are  combined  with  quantum  chemistry  and  first  principles 
dynamics,  and  3)  the  development  of  the  ab  initio  nanoreactor  which  discovers  new 
chemical  reactions  from  first  principles  simulation. 

Summary  Outcomes:  This  NSSEFF  project  resulted  in  34  peer-reviewed  publications.  It 
has  supported  13  postdoctoral  scholars/staff  scientists  and  8  graduate  students.  The  PI  has 
given  more  than  50  invited  talks  describing  the  NSSEFF-funded  research  in  the  US,  Europe 
and  Asia,  including  numerous  plenary  addresses. 

GPU  Acceleration  of  Electronic  Structure  and  Ab  Initio  Molecular  Dynamics:  The  main 
proposed  goal  of  this  project  was  to  transform  first  principles  simulation  capabilities  by 
leveraging  GPUs.  We  redesigned  algorithms  for  stream  processors  and  made  significant 
inroads  compared  to  our  first  preliminary  implementation.  It  is  now  possible  to  carry  out 
calculations  including  d  angular  momentum  basis  functions1  and  effective  core  potentials2 
(opening  up  the  possibility  of  treating  organometallic  complexes),  continuum  solvation3 
(enabling  calculations  that  include  both  explicit  and  implicit  solvent),  multireference 
wavefunction  methods4'6  (enabling  the  calculation  of  dynamics  involving  diradical  and 
other  open  shell  species),  excited  electronic  states,4'7'9  and  multiple  levels  of  precision10 
(making  consumer  grade  GPUs  a  viable  option).  We  also  introduced  a  method  to  exploit 
low  cost  methods  and  use  them  to  increase  the  size  of  the  time  steps  that  can  be  used  in 
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AIMD11  and  a  method  to  increase  the  accuracy  of  electronic  structure  methods  with  small 
basis  sets.12  We  demonstrated  the  new  groundbreaking  capabilities  through  application  to 
numerous  "firsts:”  the  first  ab  initio  molecular  dynamics  of  proteins,13  large  scale  ab  initio 
optimization  to  gauge  accuracy  in  structure  prediction/refinement,14  ab  initio  calculations 
of  electronic  excitation  energies  and  absorption  spectra  in  proteins,8  and  ab  initio  dynamics 
including  nonadiabatic  surface  crossing  effects  for  a  molecule  with  more  than  50  atoms.15 
Using  these  tools,  we  were  able  to  carry  out  numerous  studies  on  excited  state  reactivity,16' 
18  enzyme  reactions,19  and  mechanochemistry  of  polymers.20  We  also  exploited  the  ability 
to  generate  large  amounts  of  electronic  structure  information  to  parameterize  a  new 
empirical  potential  for  water.21 

Tensor  Hypercontraction:  A  key  stumbling  block  in  electronic  structure  theory  (and 
therefore  also  in  ab  initio  molecular  dynamics)  has  been  the  steep  scaling  of  wavefunction 
based  methods  for  describing  electron  correlation.  It  is  largely  because  of  the  perceived 
"scaling  wall"  that  density  functional  theory  has  become  so  successful.  DFT  scales  formally 
as  0(N4),  where  N  is  the  size  of  the  molecule  (or  more  accurately  the  number  of  basis 
functions  used  to  describe  the  electronic  density).  The  scaling  of  DFT  is  thus  the  same  as 
Hartree-Fock  (HF)  theory,  but  DFT  includes  the  electron  correlation  effects  that  HF  ignores. 
Unfortunately,  the  exchange-correlation  functional  in  DFT  remains  unknown  and  current 
approximations  have  numerous  problems  and  deficiencies  that  have  proven  difficult  to 
resolve.  Wavefunction  methods  that  include  electron  correlation  scale  as  0(N5)  or  even 
worse  -  for  example,  CCSD(T)  which  is  often  called  the  "gold  standard”  for 
thermochemistry  scales  as  0(N7).  During  the  NSSEFF  project,  we  discovered  that  it  is 
possible  to  formulate  wavefunction  methods  such  as  Moller-Plesset  perturbation 
theory  and  coupled  cluster  such  that  their  formal  scaling  is  0(N4)  -  the  same  as  DFT! 
This  discovery  was  based  on  our  investigation  into  recommendation  systems  used  by 
Netflix  and  Amazon  to  determine  what  products  should  be  suggested  to  consumers.  The 
core  idea  in  these  recommendation  systems  is  that  one  imagines  a  large  matrix  with 
consumers  as  rows  and  products  as  columns.  Then  one  asks  for  an  algorithm  that  can 
"complete”  this  matrix  on  the  basis  of  several  observations.  This  is  of  course  impossible  in 
general.  However,  it  would  be  possible  if  the  matrix  in  question  contained  very  little 
information  (for  example,  if  consumers  fell  into  groups  where  all  members  of  the  same 
group  wanted  the  same  products).  Such  information-deficient  matrices  are  technically 
called  "low-rank,"  and  the  recommendation  system  algorithms  exploit  this  structure.  We 
discovered  how  to  apply  this  idea  to  high-dimensional  tensors  and  then  how  to  use  this  for 
the  two  electron  repulsion  integrals.  We  then  discovered  that  the  same  idea  could  also  be 
used  for  the  electronic  wavefunction.  Effectively,  we  discovered  that  there  are  massive 
redundancies  in  both  the  quantum  mechanical  operators  and  wavefunctions  that  are 
encountered  in  most  molecules.  Recognizing  this  hidden  structure  and  revealing  it  allows 
one  to  decrease  the  computational  effort  by  up  to  two  powers  of  the  molecular  size.  A 
number  of  papers  demonstrate  the  idea  and  show  that  it  is  quantitatively  accurate  and 
faster  than  conventional  approaches  already  for  small  molecules  (the  accuracy  and 
efficiency  increase  as  larger  molecules  are  treated).22'31  We  also  implemented  the  method 
on  GPUs  for  further  acceleration31  and  applied  it  to  electronic  excitation  energies  with 
equation-of-motion  coupled  cluster  (EOM-CC2).26  Tensor  hypercontraction  has  breathed 
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new  life  into  wavefunction  methods  and  is  currently  being  pursued  by  numerous  other 
groups  as  well  as  our  own. 

Interactive  Ab  Initio  Molecular  Dynamics:  As  our  efforts  to  accelerate  ab  initio 
molecular  dynamics  on  GPUs  became  more  successful,  we  began  asking  ourselves  what  we 
might  be  able  to  do  that  was  qualitatively  different  from  “business  as  usual"  in 
computational  chemistry.  Sure,  larger  molecules  can  now  be  simulated  for  longer  time 
scales  and  this  is  bound  to  change  the  landscape  considerably  as  the  scope  of  problems  that 
are  amenable  to  computation  is  enlarged.  But  are  there  modes  of  use  that  were  simply 
unthinkable  before?  Thinking  about  this  led  us  to  two  ideas  -  interactive  AIMD  and  the 
nanoreactor  (see  following  paragraph).  The  main  idea  in  interactive  AIMD  is  to  allow 
the  user  to  interact  with  an  AIMD  simulation  in  real  time,  i.e.  as  the  simulation 
progresses.  We  were  successful  in  this  venture,  using  a  haptic  input  device  with  force 
feedback.32  The  user  grabs  an  atom  with  the  haptic  device  and  then  the  haptic  device 
resists  the  user  according  to  the  internal  forces  computed  by  AIMD.  Bonds  can  be  broken 
because  the  underlying  simulation  solves  the  electronic  Schrodinger  equation  at  each  time 
step.  When  the  user  tries  to  break  a  strong  bond,  more  force  is  required,  compared  to 
breaking  a  weak  bond.  This  new  technology  allows  users  to  experience  chemistry  in  full 
three  dimensions  and  with  tactile  response.  We  are  exploring  the  use  of  this  capability  for 
both  research  and  education. 

The  ab  initio  Nanoreactor:  The  second  idea  which  arose  from  thinking  about  how  to  use 
GPU-accelerated  AIMD  in  a  disruptive  manner  was  the  "ab  initio  nanoreactor.”  We  thought 
that  it  might  be  possible  to  begin  using  AIMD  as  a  means  to  discover  new  reactions.  The 
first  attempts  at  this  simply  ran  molecular  dynamics  at  high  temperatures  with  the 
molecules  confined  to  remain  within  a  sphere.  We  observed  that  reactions  could  be  seen  to 
progress  on  a  time  scale  which  was  easily  attained  (tens  of  picoseconds,  which  consumed 
less  than  a  day  of  real  time  on  a  desktop  computer).  This  led  us  to  realize  that  we  could 
accelerate  reactions  in  numerous  ways,  including  not  only  high  temperatures  but  also 
pulsing  pressure  (with  a  spherical  piston).  Once  we  discovered  reactions  in  this  way,  we 
realized  that  the  reactions  could  then  be  studied  with  traditional  hypothesis-driven 
approaches  -  finding  minimum  energy  paths  and  reaction  barriers.  This  new  tool33'34  is  a 
way  to  discover  what  reactions  are  possible  and  then  these  can  be  refined  (possibly  at  a 
more  accurate  level  of  theory  if  desired),  obtaining  rates  which  can  then  be  placed  in  a 
kinetic  model  (sometimes  called  a  set  of  master  equations).  This  represents  a  major  new 
direction  in  theoretical  chemistry,  which  can  now  begin  to  unravel  the  mechanisms 
involved  in  complex  reaction  networks.  Unlike  traditional  approaches,  the 
nanoreactor  is  a  discovery  tool.  We  are  currently  exploring  the  use  of  this  and  related 
strategies  to  discover  new  synthetic  routes,  to  expose  the  mechanisms  of  combustion 
reactions,  and  to  discover  new  catalysts  for  fundamental  chemical  reactions. 

Summary:  We  accomplished  many  of  the  goals  originally  proposed  for  the  NSSEFF  project. 
Primarily,  we  exploited  GPUs  in  ab  initio  molecular  dynamics  in  a  way  that  continues  to 
transform  the  field.  At  the  same  time,  we  would  like  to  emphasize  the  new  directions  that 
could  not  have  been  foreseen  at  the  time  the  original  proposal  was  written.  Thanks  to  the 
flexibility  of  the  NSSEFF  funding,  we  were  able  to  pursue  these  directions  that  have  proven 
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to  be  transformative.  We  developed  the  tensor  hypercontraction  formalism  that  has 
resurrected  wavefunction  theories  and  may  supplant  DFT.  We  introduced  a  new 
technology  for  interactive  ab  initio  molecular  dynamics,  which  has  now  received  funding 
through  the  STTR  program  and  could  be  the  beginning  of  a  new  effort  to  develop  molecular 
computer-aided  design  (MolCAD)  and  perhaps  ultimately  also  molecular  computer-aided- 
manufacturing.  We  also  introduced  the  ab  initio  nanoreactor  which  has  attracted  the 
attention  of  at  least  four  major  companies  so  far  and  is  being  developed  in  new  directions. 
We  are  confident  that  the  nanoreactor  strategy  will  change  our  perception  of  the  role  of 
theory  and  computation  in  chemistry  -  from  the  traditional  hypothesis-driven  approach  to 
one  emphasizing  discovery  of  chemical  pathways  and  reactions. 
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